The Design of an Auto-Tuning I/O Framework on Cray XT5 System

نویسندگان

  • Haihang You
  • Qing Liu
  • Zhiqiang Li
  • Shirley Moore
چکیده

As high performance computing (HPC) heads towards the exascale era, the computing power surges tremendously and applications will scale to hundreds of thousands cores. Consequently, the amount of data processed and generated will increase dramatically. Nowadays, a parallel shared file system is a must have for a supercomputer. To utilize I/O effectively is essential for an application to scale up. We have developed a mathematical model to describe parallel I/O activities that serves as the basis for an I/O auto-tuning infrastructure for HPC systems. Our current work is in the context of Lustre, but our ideas should be applicable to other distributed file systems. This paper explains our model, which is based on queuing theory, describes the auto-tuning process, and gives experimental results over Lustre on the Cray XT5 that show low relative error. Keywords-I/O; performance modeling; auto-tuning; queuing theory; simulation

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

An application-level parallel I/O library for Earth system models

We describe the design and implementation of an application-level parallel I/O (PIO) library for the reading and writing of distributed arrays to several common scientific data formats. PIO provides the flexibility to control the number of I/O tasks through data rearrangement to an I/O friendly decomposition. This flexibility enables reductions in per task memory usage and improvements in disk ...

متن کامل

ExPBB: A framework to explore the performance of Burst Buffer

ShaheenII supercomputer provides 268 Burst Buffer nodes based on Cray DataWarp technology. Thus, there is an extra layer between the compute nodes and the parallel filesystem by using SSDs. However, this technology is new, and many scientists try to understand and gain the maximum performance. We present an auto-tuning I/O framework called Explore the Performance of Burst Buffer. The purpose of...

متن کامل

Impact of Quad-Core Cray XT4 System and Software Stack on Scientific Computation

An upgrade from dual-core to quad-core AMD processor on the Cray XT system at the Oak Ridge National Laboratory (ORNL) Leadership Computing Facility (LCF) has resulted in significant changes in the hardware and software stack, including a deeper memory hierarchy, SIMD instructions and a multi-core aware MPI library. In this paper, we evaluate impact of a subset of these key changes on large-sca...

متن کامل

A Next-Generation Parallel File System Environment for the OLCF

When deployed in 2008/2009 the Spider system at the Oak Ridge National Laboratory’s Leadership Computing Facility (OLCF) was the world’s largest scale Lustre parallel file system. Envisioned as a shared parallel file system capable of delivering both the bandwidth and capacity requirements of the OLCF’s diverse computational environment, Spider has since become a blueprint for shared Lustre env...

متن کامل

Early Evaluation of the Cray XT5

A Cray XT5 system has recently been installed at Oak Ridge National Laboratory (ORNL). This system differs from the existing XT4 system at ORNL in its compute node architecture, utilizing two quad-core Opteron Barcelona processors instead of a single quad-core Opteron Budapest processor. It also differs in its sheer scale, having approximately 150,000 processor cores, almost 5 times as many as

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2011